Data Cleansing and Records Calculations

Part 2 of 3

This notebook follows sequentially from NOAA-CO-OPS-data in which we downloaded the latest data for a particular NOAA CO-OPS weather and tide station. The data record and corresponding metadata were written to file. Here we use those data to calculate several daily and monthly statistics and records. This is done in two steps:

1. Filter the data: We do not perform any quality assurance or quality control checks, but we do remove from the records any days missing a specified amount of data and any months missing a specified number of days of data.

2. Calculate records:

   • Daily and monthly averages
   • Record high daily and monthly averages^*
   • Record low daily and monthly averages^*
   • Average daily and monthly high
   • Lowest daily and monthly high^*
   • Record daily and monthly high^*
   • Average daily and monthly low
   • Highest daily and monthly low^*
   • Record daily and monthly low^*

Years are also noted for those records marked by an asterisk (*).

Packages and configurations

First we import the packages we need.

import pandas as pd
import xarray as xr
import calendar
import yaml
import os

By default, Python only displays warnings the first time they are thrown. Ideally, we want a code that does not throw any warnings, but it sometimes takes some trial and error to resolve the issue being warned about. So, for diagnostic purposes, we’ll set the kernel to always display warnings.

import warnings
warnings.filterwarnings('always')

Functions

Next, we define a number of functions that will come in handy later.

Helper functions

def camel(text):
    """Convert 'text' to camel case"""
    s = text.replace(',','').replace("-", " ").replace("_", " ")
    s = s.split()
    if len(text) == 0:
        return text
    return s[0].lower() + ''.join(i.capitalize() for i in s[1:])

def DOY(df):
    """Determine year day out of 366"""
    # Day of year as integer
    df['YearDay'] = df.index.day_of_year.astype(int)
    # Years that are NOT leap years
    leapInd = [not calendar.isleap(i) for i in df.index.year]
    mask = (leapInd) & (df.index.month > 2)
    # Advance by one day everything after February 28 
    df.loc[mask, 'YearDay'] += 1
    return df

Filtering data

def count_missing_hours(group, threshold=3):
    """Return True if the number of hours in a day with missing data is less
    than or equal to 'threshold' and False otherwise.
    """
    missing_hours = group.resample('1h').mean().isna().sum()
    return missing_hours <= threshold

def count_missing_days(group, threshold=2):
    """Return True if the number of days in a month with missing data is less
    than or equal to 'theshold' and False otherwise. Two tests are performed:
    missing data (NaN) and compare to the number of days in the given month.
    """
    try:
        days_in_month = pd.Period(group.index[0].strftime(format='%Y-%m-%d')).days_in_month
        missing_days = group.resample('1D').mean().isna().sum()
        missing_days_flag = missing_days <= threshold
        days_in_month_flag = days_in_month - group.resample('1D').mean().size <= threshold
        return min(missing_days_flag, days_in_month_flag)
    except IndexError:
        pass

def filter_data(data, hr_threshold=3, day_threshold=2):
    """Filter data to remove days with more than 'hr_threshold' missing hours
    of data and months with more than 'day_threshold' days of missing data.
    """
    # Filter out days missing more than <hr_threshold> hours
    filtered = data.groupby(pd.Grouper(freq='1D')).filter(lambda x: count_missing_hours(group=x, threshold=hr_threshold))
    # Filter out months missing more than <day_threshold> days
    filtered = filtered.groupby(pd.Grouper(freq='1M')).filter(lambda x: count_missing_days(group=x, threshold=day_threshold))
    return filtered

Calculate records

def daily_highs(df):
    """Daily highs"""
    return df.groupby(pd.Grouper(freq='1D')).max(numeric_only=True)

def daily_lows(df):
    """Daily lows"""
    return df.groupby(pd.Grouper(freq='1D')).min(numeric_only=True)

def daily_avgs(df, decimals=1, true_average=False):
    """Daily averages by calendar day rounded to 'decimals'. If
    'true_average' is True, all measurements from each 24-hour day will be
    used to calculate the average. Otherwise, only the maximum and minimum
    observations are used. Defaults to False (meteorological standard).
    """
    if true_average:
        results = df.groupby(pd.Grouper(freq='1D')).mean(numeric_only=True)
    else:
        dailyHighs = daily_highs(df)
        dailyLows = daily_lows(df)
        results = (dailyHighs + dailyLows) / 2
    return results.round(decimals)

def daily_avg(df, decimals=1, true_average=False):
    """Daily averages rounded to 'decimals'. If 'true_average' is True, all
    measurements from each 24-hour day will be used to calculate the daily
    average. Otherwise, only the maximum and minimum observations are used.
    Defaults to False (meteorological standard).
    """
    dailyAvgs = daily_avgs(df, decimals=decimals, true_average=true_average)
    dailyAvg = dailyAvgs.groupby('YearDay').mean(numeric_only=True).round(decimals)
    dailyAvg.index = dailyAvg.index.astype(int)
    results = xr.DataArray(dailyAvg, dims=['yearday', 'variable'])
    results.name = 'Daily Average'
    return results

def monthly_highs(df, decimals=1, true_average=False):
    """Monthly highs rounded to 'decimals'. If 'true_average' is True, all
    measurements from each 24-hour day will be used to calculate the daily
    average. Otherwise, only the maximum and minimum observations are used.
    Defaults to False (meteorological standard).
    """
    dailyAvgs = daily_avgs(df, decimals=decimals, true_average=False)
    monthHighs = dailyAvgs.groupby(pd.Grouper(freq='M')).max(numeric_only=True)
    return monthHighs
  
def monthly_lows(df, decimals=1, true_average=False):
    """Monthly lows rounded to 'decimals'. If 'true_average' is True, all
    measurements from each 24-hour day will be used to calculate the daily
    average. Otherwise, only the maximum and minimum observations are used.
    Defaults to False (meteorological standard).
    """
    dailyAvgs = daily_avgs(df, decimals=decimals, true_average=true_average)
    monthLows = dailyAvgs.groupby(pd.Grouper(freq='M')).min(numeric_only=True)
    return monthLows
    
def monthly_avg(df, decimals=1, true_average=False):
    """Monthly averages for variable 'var' rounded to 'decimals'. If
    'true_average' is True, all measurements from each 24-hour day will be
    used to calculate the daily average. Otherwise, only the maximum and
    minimum observations are used. Defaults to False (meteorological
    standard).
    """
    dailyAvgs = daily_avgs(df, decimals=decimals, true_average=true_average)
    monthlyMeans = dailyAvgs.groupby(pd.Grouper(freq='M')).mean(numeric_only=True).round(decimals)
    monthlyMeans.drop('YearDay', axis=1, inplace=True)
    monthlyAvg = monthlyMeans.groupby(monthlyMeans.index.month).mean(numeric_only=True).round(decimals)
    monthlyAvg.index = monthlyAvg.index.astype(int)
    results = xr.DataArray(monthlyAvg, dims=['month', 'variable'])
    results.name = 'Monthly Average'
    return results

def record_high_daily_avg(df, decimals=1, true_average=False):
    """Record high daily averages rounded to 'decimals'. If 'true_average'
    is True, all measurements from each 24-hour day will be used to
    calculate the daily average. Otherwise, only the maximum and minimum
    observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the records
    dailyAvgs = daily_avgs(df=df, decimals=decimals, true_average=true_average)
    recordHighDailyAvg = dailyAvgs.groupby('YearDay').max(numeric_only=True).round(decimals)
    recordHighDailyAvg.index = recordHighDailyAvg.index.astype(int)
    # Record years
    recordHighDailyAvgYear = dailyAvgs.groupby('YearDay').apply(lambda x: x.idxmax(numeric_only=True).dt.year)
    recordHighDailyAvgYear.drop('YearDay', axis=1, inplace=True)
    recordHighDailyAvgYear.index = recordHighDailyAvgYear.index.astype(int)
    recordHighDailyAvgYear.columns = [i+' Year' for i in recordHighDailyAvgYear.columns]
    # Create xarray
    results = pd.concat((recordHighDailyAvg, recordHighDailyAvgYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Record High Daily Average'
    return results
    
def record_high_monthly_avg(df, decimals=1, true_average=False):
    """Record high monthly averages rounded to 'decimals'. If
    'true_average' is True, all measurements from each 24-hour day will be
    used to calculate the daily average. Otherwise, only the maximum and
    minimum observations are used. Defaults to False (meteorological
    standard).
    """
    # Calculate the records
    dailyAvgs = daily_avgs(df, decimals=decimals, true_average=true_average)
    monthlyAvgs = dailyAvgs.groupby(pd.Grouper(freq='M')).mean(numeric_only=True).round(decimals)
    monthlyAvgs.drop('YearDay', axis=1, inplace=True)
    recordHighMonthlyAvg = monthlyAvgs.groupby(monthlyAvgs.index.month).max(numeric_only=True)
    recordHighMonthlyAvg.index = recordHighMonthlyAvg.index.astype(int)
    # Record years
    recordHighMonthlyAvgYear = monthlyAvgs.groupby(monthlyAvgs.index.month).apply(lambda x: x.idxmax(numeric_only=True).dt.year)
    recordHighMonthlyAvgYear.index = recordHighMonthlyAvgYear.index.astype(int)
    recordHighMonthlyAvgYear.columns = [i+' Year' for i in recordHighMonthlyAvgYear.columns]
    # Create xarray
    results = pd.concat((recordHighMonthlyAvg, recordHighMonthlyAvgYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Record High Monthly Average'
    return results

def record_low_daily_avg(df, decimals=1, true_average=False):
    """Record low daily averages rounded to 'decimals'.  If 'true_average'
    True, all measurements from each 24-hour day will be used to calculate
    the average. Otherwise, only the maximum and minimum observations are
    used. Defaults to False (meteorological standard)."""
    # Calculate the records
    dailyAvgs = daily_avgs(df=df, decimals=decimals, true_average=true_average)
    recordLowDailyAvg = dailyAvgs.groupby('YearDay').min(numeric_only=True).round(decimals)
    recordLowDailyAvg.index = recordLowDailyAvg.index.astype(int)
    # Record years
    recordLowDailyAvgYear = dailyAvgs.groupby('YearDay').apply(lambda x: x.idxmin(numeric_only=True).dt.year)
    recordLowDailyAvgYear.drop('YearDay', axis=1, inplace=True)
    recordLowDailyAvgYear.index = recordLowDailyAvgYear.index.astype(int)
    recordLowDailyAvgYear.columns = [i+' Year' for i in recordLowDailyAvgYear.columns]
    # Create xarray
    results = pd.concat((recordLowDailyAvg, recordLowDailyAvgYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Record Low Daily Average'
    return results

def record_low_monthly_avg(df, decimals=1, true_average=False):
    """Record low monthly averages rounded to 'decimals'. If 'true_average'
    is True, all measurements from each 24-hour day will be used to
    calculate the daily average. Otherwise, only the maximum and minimum
    observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the records
    dailyAvgs = daily_avgs(df, decimals=decimals, true_average=true_average)
    monthlyAvgs = dailyAvgs.groupby(pd.Grouper(freq='M')).mean(numeric_only=True).round(decimals)
    monthlyAvgs.drop('YearDay', axis=1, inplace=True)
    recordLowMonthlyAvg = monthlyAvgs.groupby(monthlyAvgs.index.month).min(numeric_only=True)
    recordLowMonthlyAvg.index = recordLowMonthlyAvg.index.astype(int)
    # Record years
    recordLowMonthlyAvgYear = monthlyAvgs.groupby(monthlyAvgs.index.month).apply(lambda x: x.idxmin(numeric_only=True).dt.year)
    recordLowMonthlyAvgYear.index = recordLowMonthlyAvgYear.index.astype(int)
    recordLowMonthlyAvgYear.columns = [i+' Year' for i in recordLowMonthlyAvgYear.columns]
    # Create xarray
    results = pd.concat((recordLowMonthlyAvg, recordLowMonthlyAvgYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Record Low Monthly Average'
    return results

def avg_daily_high(df, decimals=1):
    """Average daily highs rounded to 'decimals'."""        
    dailyHighs = daily_highs(df)
    results = dailyHighs.groupby('YearDay').mean(numeric_only=True).round(decimals)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Average Daily High'
    return results

def avg_monthly_high(df, decimals=1, true_average=False):
    """Average monthly highs rounded to 'decimals'. If 'true_average' is
    True, all measurements from each 24-hour day will be used to calculate
    the daily average. Otherwise, only the maximum and minimum observations
    are used. Defaults to False (meteorological standard).
    """
    monthlyHighs = monthly_highs(df, decimals=decimals, true_average=true_average)
    monthlyHighs.drop('YearDay', axis=1, inplace=True)
    avgMonthlyHighs = monthlyHighs.groupby(monthlyHighs.index.month).mean(numeric_only=True).round(decimals)
    results = xr.DataArray(avgMonthlyHighs, dims=['month', 'variable'])
    results.name = 'Average Monthly High'
    return results

def lowest_daily_high(df, decimals=1):
    """Lowest daily highs rounded to 'decimals'."""
    # Calculate the record
    dailyHighs = daily_highs(df)
    lowestHigh = dailyHighs.groupby('YearDay').min(numeric_only=True).round(decimals)
    lowestHigh.index = lowestHigh.index.astype(int)
    # Record years
    lowestHighYear = dailyHighs.groupby('YearDay').apply(lambda x: x.idxmin(numeric_only=True).dt.year)
    lowestHighYear.drop('YearDay', axis=1, inplace=True)
    lowestHighYear.index = lowestHighYear.index.astype(int)
    lowestHighYear.columns = [i+' Year' for i in lowestHighYear.columns]
    # Create xarray
    results = pd.concat((lowestHigh, lowestHighYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Lowest Daily High'
    return results
    
def lowest_monthly_high(df, decimals=1, true_average=False):
    """Lowest monthly highs rounded to 'decimals'. If 'true_average' is
    True, all measurements from each 24-hour day will be used to calculate
    the daily average. Otherwise, only the maximum and minimum observations
    are used. Defaults to False (meteorological standard).
    """
    # Calculate the record
    monthlyHighs = monthly_highs(df, decimals=decimals, true_average=true_average)
    monthlyHighs.drop('YearDay', axis=1, inplace=True)
    lowMonthlyHigh = monthlyHighs.groupby(monthlyHighs.index.month).min(numeric_only=True).round(decimals)
    lowMonthlyHigh.index = lowMonthlyHigh.index.astype(int)
    # Record years
    lowMonthlyHighYear = monthlyHighs.groupby(monthlyHighs.index.month).apply(lambda x: x.idxmin(numeric_only=True).dt.year)
    lowMonthlyHighYear.index = lowMonthlyHighYear.index.astype(int)
    lowMonthlyHighYear.columns = [i+' Year' for i in lowMonthlyHighYear.columns]
    # Create xarray
    results = pd.concat((lowMonthlyHigh, lowMonthlyHighYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Lowest Monthly High'
    return results

def record_daily_high(df, decimals=1):
    """Record daily highs rounded to 'decimal'."""
    # Calculate the record
    dailyHighs = daily_highs(df)
    recordHigh = dailyHighs.groupby('YearDay').max(numeric_only=True).round(decimals)
    recordHigh.index = recordHigh.index.astype(int)
    # Record years
    recordHighYear = dailyHighs.groupby('YearDay').apply(lambda x: x.idxmax(numeric_only=True).dt.year)
    recordHighYear.drop('YearDay', axis=1, inplace=True)
    recordHighYear.index = recordHighYear.index.astype(int)
    recordHighYear.columns = [i+' Year' for i in recordHighYear.columns]
    # Create xarray
    results = pd.concat((recordHigh, recordHighYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Record Daily High'
    return results

def record_monthly_high(df, decimals=1, true_average=False):
    """Record monthly highs rounded to 'decimals'. If 'true_average' is
    True, all measurements from each 24-hour day will be used to calculate
    the daily average. Otherwise, only the maximum and minimum observations
    are used. Defaults to False (meteorological standard).
    """
    # Calculate the record
    monthlyHighs = monthly_highs(df, decimals=decimals, true_average=true_average)
    monthlyHighs.drop('YearDay', axis=1, inplace=True)
    recordMonthlyHigh = monthlyHighs.groupby(monthlyHighs.index.month).max(numeric_only=True).round(decimals)
    recordMonthlyHigh.index = recordMonthlyHigh.index.astype(int)
    # Record years
    recordMonthlyHighYear = monthlyHighs.groupby(monthlyHighs.index.month).apply(lambda x: x.idxmax(numeric_only=True).dt.year)
    recordMonthlyHighYear.index = recordMonthlyHighYear.index.astype(int)
    recordMonthlyHighYear.columns = [i+' Year' for i in recordMonthlyHighYear.columns]
    # Create xarray
    results = pd.concat((recordMonthlyHigh, recordMonthlyHighYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Record Monthly High'
    return results

def avg_daily_low(df, decimals=1):
    """Average daily lows rounded to 'decimals'."""        
    dailyLows = daily_lows(df)
    results = dailyLows.groupby('YearDay').mean(numeric_only=True).round(decimals)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Average Daily Low'
    return results

def avg_monthly_low(df, decimals=1, true_average=False):
    """Average monthly lows rounded to 'decimals'. If 'true_average' is
    True, all measurements from each 24-hour day will be used to calculate
    the daily average. Otherwise, only the maximum and minimum observations
    are used. Defaults to False (meteorological standard).
    """
    monthlyLows = monthly_lows(df, decimals=decimals, true_average=true_average)
    monthlyLows.drop('YearDay', axis=1, inplace=True)
    avgMonthlyLows = monthlyLows.groupby(monthlyLows.index.month).mean(numeric_only=True).round(decimals)
    results = xr.DataArray(avgMonthlyLows, dims=['month', 'variable'])
    results.name = 'Average Monthly Low'
    return results

def highest_daily_low(df, decimals=1):
    """Highest daily lows rounded to 'decimals'."""
    # Calculate the record
    dailyLows = daily_lows(df)
    highestLow = dailyLows.groupby('YearDay').max(numeric_only=True).round(decimals)
    highestLow.index = highestLow.index.astype(int)
    # Record years
    highestLowYear = dailyLows.groupby('YearDay').apply(lambda x: x.idxmax(numeric_only=True).dt.year)
    highestLowYear.drop('YearDay', axis=1, inplace=True)
    highestLowYear.index = highestLowYear.index.astype(int)
    highestLowYear.columns = [i+' Year' for i in highestLowYear.columns]
    # Create xarray
    results = pd.concat((highestLow, highestLowYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Highest Daily Low'
    return results
    
def highest_monthly_low(df, decimals=1, true_average=False):
    """Highest monthly lows rounded to 'decimals'. If 'true_average' is
    True, all measurements from each 24-hour day will be used to calculate
    the daily average. Otherwise, only the maximum and minimum observations
    are used. Defaults to False (meteorological standard).
    """
    # Calculate the record
    monthlyLows = monthly_lows(df, decimals=decimals, true_average=true_average)
    monthlyLows.drop('YearDay', axis=1, inplace=True)
    highestMonthlyLow = monthlyLows.groupby(monthlyLows.index.month).max(numeric_only=True).round(decimals)
    highestMonthlyLow.index = highestMonthlyLow.index.astype(int)
    # Record years
    highestMonthlyLowYear = monthlyLows.groupby(monthlyLows.index.month).apply(lambda x: x.idxmax(numeric_only=True).dt.year)
    highestMonthlyLowYear.index = highestMonthlyLowYear.index.astype(int)
    highestMonthlyLowYear.columns = [i+' Year' for i in highestMonthlyLowYear.columns]
    # Create xarray
    results = pd.concat((highestMonthlyLow, highestMonthlyLowYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Highest Monthly Low'
    return results

def record_daily_low(df, decimals=1):
    """Record daily lows rounded to 'decimals'."""
    # Calculate the record
    dailyLows = daily_lows(df)
    recordLow = dailyLows.groupby('YearDay').min(numeric_only=True).round(decimals)
    recordLow.index = recordLow.index.astype(int)
    # Record years
    recordLowYear = dailyLows.groupby('YearDay').apply(lambda x: x.idxmin(numeric_only=True).dt.year)
    recordLowYear.drop('YearDay', axis=1, inplace=True)
    recordLowYear.index = recordLowYear.index.astype(int)
    recordLowYear.columns = [i+' Year' for i in recordLowYear.columns]
    # Create xarray
    results = pd.concat((recordLow, recordLowYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Record Daily Low'
    return results

def record_monthly_low(df, decimals=1, true_average=False):
    """Record monthly lows rounded to 'decimals'. If 'true_average' is
    True, all measurements from each 24-hour day will be used to calculate
    the daily average. Otherwise, only the maximum and minimum observations
    are used. Defaults to False (meteorological standard).
    """
    # Calculate the record
    monthlyLows = monthly_lows(df, decimals=decimals, true_average=true_average)
    monthlyLows.drop('YearDay', axis=1, inplace=True)
    recordMonthlyLow = monthlyLows.groupby(monthlyLows.index.month).min(numeric_only=True).round(decimals)
    recordMonthlyLow.index = recordMonthlyLow.index.astype(int)
    # Record years
    recordMonthlyLowYear = monthlyLows.groupby(monthlyLows.index.month).apply(lambda x: x.idxmin(numeric_only=True).dt.year)
    recordMonthlyLowYear.index = recordMonthlyLowYear.index.astype(int)
    recordMonthlyLowYear.columns = [i+' Year' for i in recordMonthlyLowYear.columns]
    # Create xarray
    results = pd.concat((recordMonthlyLow, recordMonthlyLowYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Record Monthly Low'
    return results

def number_of_years_byday(df):
    """Number of years in the historical data records by day of year."""
    numYears = pd.concat([df[[v, 'YearDay']].dropna().groupby('YearDay')\
                                        .apply(lambda x: len(x.index.year.unique())) \
                         for v in filtered_data.columns if v != 'YearDay'], axis=1)
    numYears.columns = [v for v in df.columns if v != 'YearDay']
    results = xr.DataArray(numYears, dims=['yearday', 'variable'])
    results.name = 'Number of Years'
    return results

def number_of_years_bymonth(df):
    """Number of years in the historical data records by month."""
    numYears = pd.concat([df[v].dropna().groupby(df[v].dropna().index.month)\
                                        .apply(lambda x: len(x.index.year.unique())) \
                         for v in filtered_data.columns if v != 'YearDay'], axis=1)
    numYears.columns = [v for v in df.columns if v != 'YearDay']
    results = xr.DataArray(numYears, dims=['month', 'variable'])
    results.name = 'Number of Years'
    return results

def generate_yeardays():
    return pd.date_range(start='2020-01-01',end='2020-12-31', freq='1D').strftime('%d-%b')

Data cleaning

First we need to load in the data and metadata for the desired station. As before, stationname is the custom human-readable “City, ST” string for the station. This will be used to determine the directory from which to load the data. Since we are not downloading data, we do not need the NOAA-COOPS station ID number.

stationname = 'Virginia Key, FL'

Derive the local directory name containing the data from the station name. This is the same way the directory was created when the data were downloaded.

dirname = camel(stationname)
outdir = os.path.join(os.getcwd(), dirname)

print(f"Station folder: {dirname}")
print(f"Full directory: {outdir}")

Station folder: virginiaKeyFl
Full directory: /home/climatology/virginiaKeyFl

Next, load the data and metadata.

# Metadata
with open(os.path.join(outdir, 'metadata.yml')) as m:
    meta = yaml.safe_load(m)

# Observational data
data = pd.read_csv(os.path.join(outdir, 'observational_data_record.csv.gz'),
                   index_col=f'time_{meta["tz"]}', parse_dates=True,
                   compression='infer')

Now we filter the data to remove days with more than 3 hours of missing data and months with more than 2 days of missing data. These thresholds are stored in meta and can easily be changed. We have to do this one variable at a time because this is sensor-dependent, so it takes a short while to run.

filtered_data = pd.concat([filter_data(data[var],
                                       hr_threshold=meta['hr_threshold'],
                                       day_threshold=meta['day_threshold'])
                                       for var in meta['variables']], axis=1)

Confirm that the data were filtered:

data.shape

(2466580, 2)

filtered_data.shape

(2174766, 2)

Calculate records

Now we’re ready to determine the records using all of the functions above. We’ll store these in an xarray dataset and add the appropriate metadata for convenience. But first, we need to add a day of year (DOY) column so that we can calculate daily records. We’ve used a function to do this because accounting for leap years is not trivial.

filtered_data = DOY(filtered_data)

daily_records = \
    xr.Dataset({'Daily Average': daily_avg(filtered_data),
                'Record High Daily Average': record_high_daily_avg(filtered_data),
                'Record Low Daily Average': record_low_daily_avg(filtered_data),
                'Average High': avg_daily_high(filtered_data),
                'Lowest High': lowest_daily_high(filtered_data),
                'Record High': record_daily_high(filtered_data),
                'Average Low': avg_daily_low(filtered_data),
                'Highest Low': highest_daily_low(filtered_data),
                'Record Low': record_daily_low(filtered_data),
                'Years': number_of_years_byday(filtered_data)},
               attrs={k:v for k, v in meta.items() if k not in ['outdir', 'variables', 'units']})

monthly_records = \
    xr.Dataset({'Monthly Average': monthly_avg(filtered_data),
                'Record High Monthly Average': record_high_monthly_avg(filtered_data),
                'Record Low Monthly Average': record_low_monthly_avg(filtered_data),
                'Average High': avg_monthly_high(filtered_data),
                'Lowest High': lowest_monthly_high(filtered_data),
                'Record High': record_monthly_high(filtered_data),
                'Average Low': avg_monthly_low(filtered_data),
                'Highest Low': highest_monthly_low(filtered_data),
                'Record Low': record_monthly_low(filtered_data),
                'Years': number_of_years_bymonth(filtered_data)},
               attrs={k:v for k, v in meta.items() if k not in ['outdir', 'variables', 'units']})

Add data units and time series ranges for each variable to the arrays as metadata attributes.

for k, v in meta['units'].items():
    daily_records.attrs[k+' units'] = v

for var in daily_records.coords['variable'].values:
    if 'Year' not in var:
        daily_records.attrs[var+' data range'] = \
            (filtered_data[var].dropna().index.min().strftime('%Y-%m-%d'),
             filtered_data[var].dropna().index.max().strftime('%Y-%m-%d'))

for k, v in meta['units'].items():
    monthly_records.attrs[k+' units'] = v

for var in monthly_records.coords['variable'].values:
    if 'Year' not in var:
        monthly_records.attrs[var+' data range'] = \
            (filtered_data[var].dropna().index.min().strftime('%Y-%m-%d'),
             filtered_data[var].dropna().index.max().strftime('%Y-%m-%d'))

What do we have now? Let’s take a look:

daily_records

<xarray.Dataset> Size: 120kB
Dimensions:                    (yearday: 366, variable: 4)
Coordinates:
  * yearday                    (yearday) int64 3kB 1 2 3 4 5 ... 363 364 365 366
  * variable                   (variable) object 32B 'Air Temperature' ... 'W...
Data variables:
    Daily Average              (yearday, variable) float64 12kB 71.5 nan ... nan
    Record High Daily Average  (yearday, variable) float64 12kB 78.0 ... 2.02...
    Record Low Daily Average   (yearday, variable) float64 12kB 54.4 ... 2.01...
    Average High               (yearday, variable) float64 12kB 75.0 nan ... nan
    Lowest High                (yearday, variable) float64 12kB 63.3 ... 2.01...
    Record High                (yearday, variable) float64 12kB 79.3 ... 2.02...
    Average Low                (yearday, variable) float64 12kB 67.9 nan ... nan
    Highest Low                (yearday, variable) float64 12kB 76.8 ... 2.02...
    Record Low                 (yearday, variable) float64 12kB 45.5 ... 2.01...
    Years                      (yearday, variable) float64 12kB 23.0 nan ... nan
Attributes:
    datum:                         MHHW
    day_threshold:                 2
    hr_threshold:                  3
    last_updated:                  2024-05-25 10:00:00
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    tz:                            lst
    unit_system:                   english
    Air Temperature units:         F
    Water Temperature units:       F
    Air Temperature data range:    ('1994-04-01', '2024-04-30')
    Water Temperature data range:  ('1994-04-01', '2024-04-30')

xarray.Dataset

• Dimensions:
  • yearday: 366
  • variable: 4
• Coordinates: (2)
  • yearday
    (yearday)
    int64
    1 2 3 4 5 6 ... 362 363 364 365 366

    array([  1,   2,   3, ..., 364, 365, 366])

  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
           'Water Temperature Year'], dtype=object)

• Data variables: (10)
  • Daily Average
    (yearday, variable)
    float64
    71.5 nan 72.4 nan ... nan 72.6 nan

    array([[71.5,  nan, 72.4,  nan],
           [71.8,  nan, 72.9,  nan],
           [70. ,  nan, 73. ,  nan],
           ...,
           [70.6,  nan, 72.6,  nan],
           [69.6,  nan, 72.7,  nan],
           [70.8,  nan, 72.6,  nan]])

  • Record High Daily Average
    (yearday, variable)
    float64
    78.0 2.022e+03 ... 80.5 2.021e+03

    array([[  78. , 2022. ,   80.4, 2022. ],
           [  77.8, 2022. ,   80.2, 2022. ],
           [  78. , 2015. ,   80.7, 2017. ],
           ...,
           [  79.4, 2015. ,   82.6, 2016. ],
           [  79.5, 2015. ,   81.9, 2016. ],
           [  79.2, 2015. ,   80.5, 2021. ]])

  • Record Low Daily Average
    (yearday, variable)
    float64
    54.4 2.001e+03 ... 66.1 2.01e+03

    array([[  54.4, 2001. ,   66.4, 2011. ],
           [  56.6, 2010. ,   66.8, 2011. ],
           [  51.7, 2012. ,   67.3, 2011. ],
           ...,
           [  58.7, 2009. ,   66.6, 1995. ],
           [  54.8, 2000. ,   66.4, 2010. ],
           [  49.9, 2000. ,   66.1, 2010. ]])

  • Average High
    (yearday, variable)
    float64
    75.0 nan 73.7 nan ... nan 73.9 nan

    array([[75. ,  nan, 73.7,  nan],
           [75.1,  nan, 74.2,  nan],
           [73.9,  nan, 74.3,  nan],
           ...,
           [74.4,  nan, 74.1,  nan],
           [73.2,  nan, 74.1,  nan],
           [74.3,  nan, 73.9,  nan]])

  • Lowest High
    (yearday, variable)
    float64
    63.3 2.001e+03 ... 67.8 2.01e+03

    array([[  63.3, 2001. ,   67.8, 2011. ],
           [  64.2, 2010. ,   68. , 2011. ],
           [  57. , 2012. ,   68.9, 2011. ],
           ...,
           [  67.1, 2010. ,   69.4, 2003. ],
           [  59.5, 2000. ,   68.4, 2010. ],
           [  55.6, 2000. ,   67.8, 2010. ]])

  • Record High
    (yearday, variable)
    float64
    79.3 2.022e+03 ... 81.7 2.021e+03

    array([[  79.3, 2022. ,   81.1, 2022. ],
           [  79.3, 2022. ,   81. , 2022. ],
           [  79.2, 2019. ,   81.5, 2017. ],
           ...,
           [  80.6, 2015. ,   83.3, 2016. ],
           [  80.6, 2013. ,   83.1, 2016. ],
           [  80.1, 2015. ,   81.7, 2021. ]])

  • Average Low
    (yearday, variable)
    float64
    67.9 nan 71.1 nan ... nan 71.4 nan

    array([[67.9,  nan, 71.1,  nan],
           [68.5,  nan, 71.6,  nan],
           [66.1,  nan, 71.7,  nan],
           ...,
           [66.7,  nan, 71.2,  nan],
           [66.1,  nan, 71.4,  nan],
           [67.3,  nan, 71.4,  nan]])

  • Highest Low
    (yearday, variable)
    float64
    76.8 2.022e+03 ... 79.3 2.021e+03

    array([[  76.8, 2022. ,   79.7, 2022. ],
           [  76.3, 2017. ,   79.5, 2022. ],
           [  77. , 2015. ,   79.9, 2017. ],
           ...,
           [  78.3, 2015. ,   81.9, 2016. ],
           [  78.4, 2015. ,   80.8, 2016. ],
           [  78.3, 2015. ,   79.3, 2021. ]])

  • Record Low
    (yearday, variable)
    float64
    45.5 2.001e+03 ... 64.4 2.01e+03

    array([[  45.5, 2001. ,   63.9, 2001. ],
           [  49.1, 2010. ,   64.4, 2001. ],
           [  46.4, 2012. ,   65.5, 2001. ],
           ...,
           [  50.1, 2009. ,   62.2, 2010. ],
           [  50. , 2000. ,   64.4, 2010. ],
           [  44.2, 2000. ,   64.4, 2010. ]])

  • Years
    (yearday, variable)
    float64
    23.0 nan 24.0 nan ... nan 24.0 nan

    array([[23., nan, 24., nan],
           [23., nan, 24., nan],
           [23., nan, 24., nan],
           ...,
           [24., nan, 24., nan],
           [24., nan, 24., nan],
           [23., nan, 24., nan]])

• Indexes: (2)
  • yearday
    PandasIndex

    PandasIndex(Int64Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
                ...
                357, 358, 359, 360, 361, 362, 363, 364, 365, 366],
               dtype='int64', name='yearday', length=366))

  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
           'Water Temperature Year'],
          dtype='object', name='variable'))

• Attributes: (12)

  datum :

  MHHW

  day_threshold :

  2

  hr_threshold :

  3

  last_updated :

  2024-05-25 10:00:00

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  F

  Water Temperature units :

  F

  Air Temperature data range :

  ('1994-04-01', '2024-04-30')

  Water Temperature data range :

  ('1994-04-01', '2024-04-30')

monthly_records

<xarray.Dataset> Size: 4kB
Dimensions:                      (month: 12, variable: 4)
Coordinates:
  * month                        (month) int64 96B 1 2 3 4 5 6 7 8 9 10 11 12
  * variable                     (variable) object 32B 'Air Temperature' ... ...
Data variables:
    Monthly Average              (month, variable) float64 384B 68.7 nan ... nan
    Record High Monthly Average  (month, variable) float64 384B 72.6 ... 2.01...
    Record Low Monthly Average   (month, variable) float64 384B 63.0 ... 2.01...
    Average High                 (month, variable) float64 384B 76.0 nan ... nan
    Lowest High                  (month, variable) float64 384B 73.0 ... 2.00...
    Record High                  (month, variable) float64 384B 78.0 ... 2.01...
    Average Low                  (month, variable) float64 384B 55.6 nan ... nan
    Highest Low                  (month, variable) float64 384B 63.5 ... 2.01...
    Record Low                   (month, variable) float64 384B 48.3 ... 2.01...
    Years                        (month, variable) float64 384B 23.0 nan ... nan
Attributes:
    datum:                         MHHW
    day_threshold:                 2
    hr_threshold:                  3
    last_updated:                  2024-05-25 10:00:00
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    tz:                            lst
    unit_system:                   english
    Air Temperature units:         F
    Water Temperature units:       F
    Air Temperature data range:    ('1994-04-01', '2024-04-30')
    Water Temperature data range:  ('1994-04-01', '2024-04-30')

xarray.Dataset

• Dimensions:
  • month: 12
  • variable: 4
• Coordinates: (2)
  • month
    (month)
    int64
    1 2 3 4 5 6 7 8 9 10 11 12

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])

  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
           'Water Temperature Year'], dtype=object)

• Data variables: (10)
  • Monthly Average
    (month, variable)
    float64
    68.7 nan 71.6 nan ... nan 74.0 nan

    array([[68.7,  nan, 71.6,  nan],
           [70.8,  nan, 72.8,  nan],
           [72.3,  nan, 75. ,  nan],
           [75.6,  nan, 78.5,  nan],
           [78.7,  nan, 82. ,  nan],
           [81.5,  nan, 85.2,  nan],
           [82.9,  nan, 87.1,  nan],
           [83.2,  nan, 87.3,  nan],
           [82. ,  nan, 85.7,  nan],
           [79.6,  nan, 82.1,  nan],
           [75. ,  nan, 77.4,  nan],
           [71.4,  nan, 74. ,  nan]])

  • Record High Monthly Average
    (month, variable)
    float64
    72.6 2.013e+03 ... 82.5 2.016e+03

    array([[  72.6, 2013. ,   78.9, 2017. ],
           [  74.9, 2018. ,   76.3, 2021. ],
           [  77.6, 2003. ,   80. , 2003. ],
           [  79.4, 2020. ,   83.4, 2020. ],
           [  80.7, 1995. ,   84.1, 2021. ],
           [  83.6, 2010. ,   87.6, 2010. ],
           [  85. , 2023. ,   89.5, 2023. ],
           [  85.9, 2022. ,   90.1, 2021. ],
           [  82.7, 2017. ,   89.5, 2021. ],
           [  81.2, 2020. ,   85.9, 2021. ],
           [  78.6, 2015. ,   81.5, 2016. ],
           [  76.9, 2015. ,   82.5, 2016. ]])

  • Record Low Monthly Average
    (month, variable)
    float64
    63.0 2.001e+03 ... 68.2 2.01e+03

    array([[  63. , 2001. ,   67.5, 2001. ],
           [  65.5, 1996. ,   67.7, 2005. ],
           [  66.1, 2010. ,   69.2, 2010. ],
           [  72.8, 2004. ,   75.1, 2004. ],
           [  77. , 2013. ,   78.6, 2001. ],
           [  79.8, 2014. ,   83.3, 2002. ],
           [  81. , 2013. ,   84.5, 2013. ],
           [  81.8, 1994. ,   85.4, 1995. ],
           [  80.6, 2001. ,   82.7, 2004. ],
           [  77.5, 2000. ,   79.6, 2000. ],
           [  71.4, 2012. ,   74.6, 2012. ],
           [  62.1, 2010. ,   68.2, 2010. ]])

  • Average High
    (month, variable)
    float64
    76.0 nan 75.4 nan ... nan 77.6 nan

    array([[76. ,  nan, 75.4,  nan],
           [76.5,  nan, 76.7,  nan],
           [78.5,  nan, 79.2,  nan],
           [80.8,  nan, 81.9,  nan],
           [82.5,  nan, 85.1,  nan],
           [84.8,  nan, 87.9,  nan],
           [85.8,  nan, 89.1,  nan],
           [85.7,  nan, 89.4,  nan],
           [85.1,  nan, 88. ,  nan],
           [83.8,  nan, 85.6,  nan],
           [79.7,  nan, 80.6,  nan],
           [77.5,  nan, 77.6,  nan]])

  • Lowest High
    (month, variable)
    float64
    73.0 2.011e+03 ... 73.0 2.003e+03

    array([[  73. , 2011. ,   70.5, 2011. ],
           [  74.2, 2000. ,   73.7, 2016. ],
           [  74.2, 2010. ,   73.3, 2010. ],
           [  77.3, 2004. ,   78.4, 2010. ],
           [  80.8, 2014. ,   82.8, 2013. ],
           [  82.8, 2014. ,   85.3, 1996. ],
           [  84.2, 2012. ,   86.4, 2013. ],
           [  84. , 2003. ,   87.2, 2000. ],
           [  83.9, 2000. ,   85.2, 1997. ],
           [  81. , 2010. ,   82.7, 2004. ],
           [  76.9, 2012. ,   76.8, 2001. ],
           [  72.5, 2010. ,   73. , 2003. ]])

  • Record High
    (month, variable)
    float64
    78.0 2.015e+03 ... 84.6 2.016e+03

    array([[  78. , 2015. ,   81.7, 2017. ],
           [  78.6, 2021. ,   81.5, 2021. ],
           [  82.8, 2003. ,   83.2, 2021. ],
           [  85.8, 2020. ,   85.8, 2020. ],
           [  85.2, 1995. ,   87.7, 2021. ],
           [  87.6, 2009. ,   90.4, 2010. ],
           [  88.7, 2018. ,   92. , 2021. ],
           [  88.5, 2022. ,   92.2, 2021. ],
           [  86.7, 2021. ,   91.2, 2021. ],
           [  86.8, 2023. ,   89. , 2016. ],
           [  82. , 2020. ,   85. , 2020. ],
           [  79.6, 1994. ,   84.6, 2016. ]])

  • Average Low
    (month, variable)
    float64
    55.6 nan 67.6 nan ... nan 70.2 nan

    array([[55.6,  nan, 67.6,  nan],
           [59.4,  nan, 68.9,  nan],
           [63.3,  nan, 70.9,  nan],
           [68.3,  nan, 74.6,  nan],
           [73.8,  nan, 78.1,  nan],
           [77.6,  nan, 82.2,  nan],
           [79. ,  nan, 84.5,  nan],
           [79.3,  nan, 84.4,  nan],
           [78.2,  nan, 83. ,  nan],
           [72.7,  nan, 77.9,  nan],
           [66. ,  nan, 74.2,  nan],
           [59.2,  nan, 70.2,  nan]])

  • Highest Low
    (month, variable)
    float64
    63.5 2.013e+03 ... 79.6 2.016e+03

    array([[  63.5, 2013. ,   75.9, 2017. ],
           [  70. , 2018. ,   73.4, 2023. ],
           [  72. , 1997. ,   75.4, 1997. ],
           [  72.6, 2015. ,   81.1, 2020. ],
           [  77.1, 2003. ,   80.4, 1994. ],
           [  80.8, 2004. ,   85.2, 2004. ],
           [  82.3, 2022. ,   87.2, 2023. ],
           [  83.6, 2022. ,   87.2, 2021. ],
           [  79.8, 2009. ,   87.1, 2021. ],
           [  77.8, 1995. ,   83.4, 2021. ],
           [  74.4, 2020. ,   80. , 2016. ],
           [  70.5, 2015. ,   79.6, 2016. ]])

  • Record Low
    (month, variable)
    float64
    48.3 1.997e+03 ... 63.7 2.01e+03

    array([[  48.3, 1997. ,   64. , 2003. ],
           [  47.9, 1996. ,   63.2, 2005. ],
           [  55.1, 1996. ,   64.8, 2010. ],
           [  61.2, 2009. ,   69.4, 2003. ],
           [  67.9, 1999. ,   74.1, 2001. ],
           [  75.1, 1995. ,   80. , 1996. ],
           [  76.1, 2013. ,   80.9, 2013. ],
           [  76.1, 1996. ,   81. , 2012. ],
           [  74.3, 2001. ,   79.5, 2004. ],
           [  64.6, 2005. ,   73.2, 2005. ],
           [  57.4, 2006. ,   70.8, 2000. ],
           [  48.8, 2010. ,   63.7, 2010. ]])

  • Years
    (month, variable)
    float64
    23.0 nan 24.0 nan ... nan 24.0 nan

    array([[23., nan, 24., nan],
           [23., nan, 21., nan],
           [24., nan, 25., nan],
           [24., nan, 25., nan],
           [21., nan, 23., nan],
           [20., nan, 23., nan],
           [25., nan, 25., nan],
           [24., nan, 25., nan],
           [24., nan, 24., nan],
           [23., nan, 23., nan],
           [23., nan, 24., nan],
           [24., nan, 24., nan]])

• Indexes: (2)
  • month
    PandasIndex

    PandasIndex(Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], dtype='int64', name='month'))

  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
           'Water Temperature Year'],
          dtype='object', name='variable'))

• Attributes: (12)

  datum :

  MHHW

  day_threshold :

  2

  hr_threshold :

  3

  last_updated :

  2024-05-25 10:00:00

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  F

  Water Temperature units :

  F

  Air Temperature data range :

  ('1994-04-01', '2024-04-30')

  Water Temperature data range :

  ('1994-04-01', '2024-04-30')

How are these stored? Let’s consider the monthly stats. Each statistic is its own variable within the dataset. Take Record High for example:

monthly_records['Record High']

<xarray.DataArray 'Record High' (month: 12, variable: 4)> Size: 384B
array([[  78. , 2015. ,   81.7, 2017. ],
       [  78.6, 2021. ,   81.5, 2021. ],
       [  82.8, 2003. ,   83.2, 2021. ],
       [  85.8, 2020. ,   85.8, 2020. ],
       [  85.2, 1995. ,   87.7, 2021. ],
       [  87.6, 2009. ,   90.4, 2010. ],
       [  88.7, 2018. ,   92. , 2021. ],
       [  88.5, 2022. ,   92.2, 2021. ],
       [  86.7, 2021. ,   91.2, 2021. ],
       [  86.8, 2023. ,   89. , 2016. ],
       [  82. , 2020. ,   85. , 2020. ],
       [  79.6, 1994. ,   84.6, 2016. ]])
Coordinates:
  * month     (month) int64 96B 1 2 3 4 5 6 7 8 9 10 11 12
  * variable  (variable) object 32B 'Air Temperature' ... 'Water Temperature ...

xarray.DataArray

'Record High'

• month: 12
• variable: 4

• 78.0 2.015e+03 81.7 2.017e+03 78.6 ... 79.6 1.994e+03 84.6 2.016e+03

  array([[  78. , 2015. ,   81.7, 2017. ],
         [  78.6, 2021. ,   81.5, 2021. ],
         [  82.8, 2003. ,   83.2, 2021. ],
         [  85.8, 2020. ,   85.8, 2020. ],
         [  85.2, 1995. ,   87.7, 2021. ],
         [  87.6, 2009. ,   90.4, 2010. ],
         [  88.7, 2018. ,   92. , 2021. ],
         [  88.5, 2022. ,   92.2, 2021. ],
         [  86.7, 2021. ,   91.2, 2021. ],
         [  86.8, 2023. ,   89. , 2016. ],
         [  82. , 2020. ,   85. , 2020. ],
         [  79.6, 1994. ,   84.6, 2016. ]])

• Coordinates: (2)
  • month
    (month)
    int64
    1 2 3 4 5 6 7 8 9 10 11 12

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])

  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
           'Water Temperature Year'], dtype=object)

• Indexes: (2)
  • month
    PandasIndex

    PandasIndex(Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], dtype='int64', name='month'))

  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
           'Water Temperature Year'],
          dtype='object', name='variable'))

• Attributes: (0)

Here, the rows are months and the columns are the records or corresponding year. Let’s see what the variables are:

monthly_records['Record High'].coords['variable']

<xarray.DataArray 'variable' (variable: 4)> Size: 32B
array(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
       'Water Temperature Year'], dtype=object)
Coordinates:
  * variable  (variable) object 32B 'Air Temperature' ... 'Water Temperature ...

xarray.DataArray

'variable'

• variable: 4

• 'Air Temperature' 'Air Temperature Year' ... 'Water Temperature Year'

  array(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
         'Water Temperature Year'], dtype=object)

• Coordinates: (1)
  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
           'Water Temperature Year'], dtype=object)

• Indexes: (1)
  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Air Temperature Year', 'Water Temperature',
           'Water Temperature Year'],
          dtype='object', name='variable'))

• Attributes: (0)

Alternatively, we can select a specific variable and see all of its stats (converting to a dataframe makes it easier to see):

monthly_records.sel(variable='Air Temperature').to_dataframe().drop('variable', axis=1)

	Monthly Average	Record High Monthly Average	Record Low Monthly Average	Average High	Lowest High	Record High	Average Low	Highest Low	Record Low	Years
month
1	68.7	72.6	63.0	76.0	73.0	78.0	55.6	63.5	48.3	23.0
2	70.8	74.9	65.5	76.5	74.2	78.6	59.4	70.0	47.9	23.0
3	72.3	77.6	66.1	78.5	74.2	82.8	63.3	72.0	55.1	24.0
4	75.6	79.4	72.8	80.8	77.3	85.8	68.3	72.6	61.2	24.0
5	78.7	80.7	77.0	82.5	80.8	85.2	73.8	77.1	67.9	21.0
6	81.5	83.6	79.8	84.8	82.8	87.6	77.6	80.8	75.1	20.0
7	82.9	85.0	81.0	85.8	84.2	88.7	79.0	82.3	76.1	25.0
8	83.2	85.9	81.8	85.7	84.0	88.5	79.3	83.6	76.1	24.0
9	82.0	82.7	80.6	85.1	83.9	86.7	78.2	79.8	74.3	24.0
10	79.6	81.2	77.5	83.8	81.0	86.8	72.7	77.8	64.6	23.0
11	75.0	78.6	71.4	79.7	76.9	82.0	66.0	74.4	57.4	23.0
12	71.4	76.9	62.1	77.5	72.5	79.6	59.2	70.5	48.8	24.0

Reorganize

For the sake of convenience later, let’s rearrange these data arrays before saving them. It will be more useful to have records and years as data variables instead of a dimension, but we have to do some renaming in order to pull this off.

First, separate the records and years into smaller xarrays:

day_records = daily_records.sel(variable=[i for i in daily_records.coords['variable'].values if 'Year' not in i])
day_years = daily_records.sel(variable=[i for i in daily_records.coords['variable'].values if 'Year' in i])

mon_records = monthly_records.sel(variable=[i for i in monthly_records.coords['variable'].values if 'Year' not in i])
mon_years = monthly_records.sel(variable=[i for i in monthly_records.coords['variable'].values if 'Year' in i])

Next, add “Year” to all of the variable names and remove it from the coordinate name:

day_years = day_years.rename_vars({i:i+' Year' for i in day_years.data_vars})
day_years.coords['variable'] = [i.removesuffix(' Year') for i in day_years.coords['variable'].values]

mon_years = mon_years.rename_vars({i:i+' Year' for i in mon_years.data_vars})
mon_years.coords['variable'] = [i.removesuffix(' Year') for i in mon_years.coords['variable'].values]

Now we can merge these two xarrays together, rearrange the order of the variables, and drop those that do not contain a year, such as daily average.

daily_records = xr.merge([day_records, day_years])
daily_records = daily_records[[item for items in zip(day_records.data_vars, day_years.data_vars) for item in items]]
daily_records = daily_records.drop_vars([x for x in daily_records.data_vars if daily_records[x].isnull().all()])

monthly_records = xr.merge([mon_records, mon_years])
monthly_records = monthly_records[[item for items in zip(mon_records.data_vars, mon_years.data_vars) for item in items]]
monthly_records = monthly_records.drop_vars([x for x in monthly_records.data_vars if monthly_records[x].isnull().all()])

monthly_records

<xarray.Dataset> Size: 3kB
Dimensions:                           (month: 12, variable: 2)
Coordinates:
  * month                             (month) int64 96B 1 2 3 4 5 ... 9 10 11 12
  * variable                          (variable) object 16B 'Air Temperature'...
Data variables: (12/16)
    Monthly Average                   (month, variable) float64 192B 68.7 ......
    Record High Monthly Average       (month, variable) float64 192B 72.6 ......
    Record High Monthly Average Year  (month, variable) float64 192B 2.013e+0...
    Record Low Monthly Average        (month, variable) float64 192B 63.0 ......
    Record Low Monthly Average Year   (month, variable) float64 192B 2.001e+0...
    Average High                      (month, variable) float64 192B 76.0 ......
    ...                                ...
    Average Low                       (month, variable) float64 192B 55.6 ......
    Highest Low                       (month, variable) float64 192B 63.5 ......
    Highest Low Year                  (month, variable) float64 192B 2.013e+0...
    Record Low                        (month, variable) float64 192B 48.3 ......
    Record Low Year                   (month, variable) float64 192B 1.997e+0...
    Years                             (month, variable) float64 192B 23.0 ......
Attributes:
    datum:                         MHHW
    day_threshold:                 2
    hr_threshold:                  3
    last_updated:                  2024-05-25 10:00:00
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    tz:                            lst
    unit_system:                   english
    Air Temperature units:         F
    Water Temperature units:       F
    Air Temperature data range:    ('1994-04-01', '2024-04-30')
    Water Temperature data range:  ('1994-04-01', '2024-04-30')

xarray.Dataset

• Dimensions:
  • month: 12
  • variable: 2
• Coordinates: (2)
  • month
    (month)
    int64
    1 2 3 4 5 6 7 8 9 10 11 12

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])

  • variable
    (variable)
    object
    'Air Temperature' 'Water Tempera...

    array(['Air Temperature', 'Water Temperature'], dtype=object)

• Data variables: (16)
  • Monthly Average
    (month, variable)
    float64
    68.7 71.6 70.8 ... 77.4 71.4 74.0

    array([[68.7, 71.6],
           [70.8, 72.8],
           [72.3, 75. ],
           [75.6, 78.5],
           [78.7, 82. ],
           [81.5, 85.2],
           [82.9, 87.1],
           [83.2, 87.3],
           [82. , 85.7],
           [79.6, 82.1],
           [75. , 77.4],
           [71.4, 74. ]])

  • Record High Monthly Average
    (month, variable)
    float64
    72.6 78.9 74.9 ... 81.5 76.9 82.5

    array([[72.6, 78.9],
           [74.9, 76.3],
           [77.6, 80. ],
           [79.4, 83.4],
           [80.7, 84.1],
           [83.6, 87.6],
           [85. , 89.5],
           [85.9, 90.1],
           [82.7, 89.5],
           [81.2, 85.9],
           [78.6, 81.5],
           [76.9, 82.5]])

  • Record High Monthly Average Year
    (month, variable)
    float64
    2.013e+03 2.017e+03 ... 2.016e+03

    array([[2013., 2017.],
           [2018., 2021.],
           [2003., 2003.],
           [2020., 2020.],
           [1995., 2021.],
           [2010., 2010.],
           [2023., 2023.],
           [2022., 2021.],
           [2017., 2021.],
           [2020., 2021.],
           [2015., 2016.],
           [2015., 2016.]])

  • Record Low Monthly Average
    (month, variable)
    float64
    63.0 67.5 65.5 ... 74.6 62.1 68.2

    array([[63. , 67.5],
           [65.5, 67.7],
           [66.1, 69.2],
           [72.8, 75.1],
           [77. , 78.6],
           [79.8, 83.3],
           [81. , 84.5],
           [81.8, 85.4],
           [80.6, 82.7],
           [77.5, 79.6],
           [71.4, 74.6],
           [62.1, 68.2]])

  • Record Low Monthly Average Year
    (month, variable)
    float64
    2.001e+03 2.001e+03 ... 2.01e+03

    array([[2001., 2001.],
           [1996., 2005.],
           [2010., 2010.],
           [2004., 2004.],
           [2013., 2001.],
           [2014., 2002.],
           [2013., 2013.],
           [1994., 1995.],
           [2001., 2004.],
           [2000., 2000.],
           [2012., 2012.],
           [2010., 2010.]])

  • Average High
    (month, variable)
    float64
    76.0 75.4 76.5 ... 80.6 77.5 77.6

    array([[76. , 75.4],
           [76.5, 76.7],
           [78.5, 79.2],
           [80.8, 81.9],
           [82.5, 85.1],
           [84.8, 87.9],
           [85.8, 89.1],
           [85.7, 89.4],
           [85.1, 88. ],
           [83.8, 85.6],
           [79.7, 80.6],
           [77.5, 77.6]])

  • Lowest High
    (month, variable)
    float64
    73.0 70.5 74.2 ... 76.8 72.5 73.0

    array([[73. , 70.5],
           [74.2, 73.7],
           [74.2, 73.3],
           [77.3, 78.4],
           [80.8, 82.8],
           [82.8, 85.3],
           [84.2, 86.4],
           [84. , 87.2],
           [83.9, 85.2],
           [81. , 82.7],
           [76.9, 76.8],
           [72.5, 73. ]])

  • Lowest High Year
    (month, variable)
    float64
    2.011e+03 2.011e+03 ... 2.003e+03

    array([[2011., 2011.],
           [2000., 2016.],
           [2010., 2010.],
           [2004., 2010.],
           [2014., 2013.],
           [2014., 1996.],
           [2012., 2013.],
           [2003., 2000.],
           [2000., 1997.],
           [2010., 2004.],
           [2012., 2001.],
           [2010., 2003.]])

  • Record High
    (month, variable)
    float64
    78.0 81.7 78.6 ... 85.0 79.6 84.6

    array([[78. , 81.7],
           [78.6, 81.5],
           [82.8, 83.2],
           [85.8, 85.8],
           [85.2, 87.7],
           [87.6, 90.4],
           [88.7, 92. ],
           [88.5, 92.2],
           [86.7, 91.2],
           [86.8, 89. ],
           [82. , 85. ],
           [79.6, 84.6]])

  • Record High Year
    (month, variable)
    float64
    2.015e+03 2.017e+03 ... 2.016e+03

    array([[2015., 2017.],
           [2021., 2021.],
           [2003., 2021.],
           [2020., 2020.],
           [1995., 2021.],
           [2009., 2010.],
           [2018., 2021.],
           [2022., 2021.],
           [2021., 2021.],
           [2023., 2016.],
           [2020., 2020.],
           [1994., 2016.]])

  • Average Low
    (month, variable)
    float64
    55.6 67.6 59.4 ... 74.2 59.2 70.2

    array([[55.6, 67.6],
           [59.4, 68.9],
           [63.3, 70.9],
           [68.3, 74.6],
           [73.8, 78.1],
           [77.6, 82.2],
           [79. , 84.5],
           [79.3, 84.4],
           [78.2, 83. ],
           [72.7, 77.9],
           [66. , 74.2],
           [59.2, 70.2]])

  • Highest Low
    (month, variable)
    float64
    63.5 75.9 70.0 ... 80.0 70.5 79.6

    array([[63.5, 75.9],
           [70. , 73.4],
           [72. , 75.4],
           [72.6, 81.1],
           [77.1, 80.4],
           [80.8, 85.2],
           [82.3, 87.2],
           [83.6, 87.2],
           [79.8, 87.1],
           [77.8, 83.4],
           [74.4, 80. ],
           [70.5, 79.6]])

  • Highest Low Year
    (month, variable)
    float64
    2.013e+03 2.017e+03 ... 2.016e+03

    array([[2013., 2017.],
           [2018., 2023.],
           [1997., 1997.],
           [2015., 2020.],
           [2003., 1994.],
           [2004., 2004.],
           [2022., 2023.],
           [2022., 2021.],
           [2009., 2021.],
           [1995., 2021.],
           [2020., 2016.],
           [2015., 2016.]])

  • Record Low
    (month, variable)
    float64
    48.3 64.0 47.9 ... 70.8 48.8 63.7

    array([[48.3, 64. ],
           [47.9, 63.2],
           [55.1, 64.8],
           [61.2, 69.4],
           [67.9, 74.1],
           [75.1, 80. ],
           [76.1, 80.9],
           [76.1, 81. ],
           [74.3, 79.5],
           [64.6, 73.2],
           [57.4, 70.8],
           [48.8, 63.7]])

  • Record Low Year
    (month, variable)
    float64
    1.997e+03 2.003e+03 ... 2.01e+03

    array([[1997., 2003.],
           [1996., 2005.],
           [1996., 2010.],
           [2009., 2003.],
           [1999., 2001.],
           [1995., 1996.],
           [2013., 2013.],
           [1996., 2012.],
           [2001., 2004.],
           [2005., 2005.],
           [2006., 2000.],
           [2010., 2010.]])

  • Years
    (month, variable)
    float64
    23.0 24.0 23.0 ... 24.0 24.0 24.0

    array([[23., 24.],
           [23., 21.],
           [24., 25.],
           [24., 25.],
           [21., 23.],
           [20., 23.],
           [25., 25.],
           [24., 25.],
           [24., 24.],
           [23., 23.],
           [23., 24.],
           [24., 24.]])

• Indexes: (2)
  • month
    PandasIndex

    PandasIndex(Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], dtype='int64', name='month'))

  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Water Temperature'], dtype='object', name='variable'))

• Attributes: (12)

  datum :

  MHHW

  day_threshold :

  2

  hr_threshold :

  3

  last_updated :

  2024-05-25 10:00:00

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  F

  Water Temperature units :

  F

  Air Temperature data range :

  ('1994-04-01', '2024-04-30')

  Water Temperature data range :

  ('1994-04-01', '2024-04-30')

Finally, let’s convert years to integers since we do not need decimal years.

daily_records[[i for i in daily_records.data_vars if "Year" in i]] = \
    daily_records[[i for i in daily_records.data_vars if "Year" in i]].astype(int)

monthly_records[[i for i in monthly_records.data_vars if "Year" in i]] = \
    monthly_records[[i for i in monthly_records.data_vars if "Year" in i]].astype(int)

‘yearday’ is not intuitive, so we can change it to calendar day instead and rename the coordinate. Similarly, we can use month names instead of numbers for the sake of clarity.

daily_records.coords['yearday'] = pd.date_range(start='2020-01-01', end='2020-12-31', freq='1D').strftime('%d-%b')
daily_records = daily_records.rename({'yearday':'Date'})

monthly_records.coords['month'] = pd.date_range(start='2020-01-01', end='2020-12-31', freq='1m').strftime('%b')
monthly_records = monthly_records.rename({'month': 'Month'})

Now take a look at the final products

daily_records

<xarray.Dataset> Size: 97kB
Dimensions:                         (Date: 366, variable: 2)
Coordinates:
  * variable                        (variable) object 16B 'Air Temperature' '...
  * Date                            (Date) object 3kB '01-Jan' ... '31-Dec'
Data variables: (12/16)
    Daily Average                   (Date, variable) float64 6kB 71.5 ... 72.6
    Record High Daily Average       (Date, variable) float64 6kB 78.0 ... 80.5
    Record High Daily Average Year  (Date, variable) int64 6kB 2022 ... 2021
    Record Low Daily Average        (Date, variable) float64 6kB 54.4 ... 66.1
    Record Low Daily Average Year   (Date, variable) int64 6kB 2001 ... 2010
    Average High                    (Date, variable) float64 6kB 75.0 ... 73.9
    ...                              ...
    Average Low                     (Date, variable) float64 6kB 67.9 ... 71.4
    Highest Low                     (Date, variable) float64 6kB 76.8 ... 79.3
    Highest Low Year                (Date, variable) int64 6kB 2022 ... 2021
    Record Low                      (Date, variable) float64 6kB 45.5 ... 64.4
    Record Low Year                 (Date, variable) int64 6kB 2001 ... 2010
    Years                           (Date, variable) int64 6kB 23 24 ... 23 24
Attributes:
    datum:                         MHHW
    day_threshold:                 2
    hr_threshold:                  3
    last_updated:                  2024-05-25 10:00:00
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    tz:                            lst
    unit_system:                   english
    Air Temperature units:         F
    Water Temperature units:       F
    Air Temperature data range:    ('1994-04-01', '2024-04-30')
    Water Temperature data range:  ('1994-04-01', '2024-04-30')

xarray.Dataset

• Dimensions:
  • Date: 366
  • variable: 2
• Coordinates: (2)
  • variable
    (variable)
    object
    'Air Temperature' 'Water Tempera...

    array(['Air Temperature', 'Water Temperature'], dtype=object)

  • Date
    (Date)
    object
    '01-Jan' '02-Jan' ... '31-Dec'

    array(['01-Jan', '02-Jan', '03-Jan', ..., '29-Dec', '30-Dec', '31-Dec'],
          dtype=object)

• Data variables: (16)
  • Daily Average
    (Date, variable)
    float64
    71.5 72.4 71.8 ... 72.7 70.8 72.6

    array([[71.5, 72.4],
           [71.8, 72.9],
           [70. , 73. ],
           [69.3, 72.7],
           [68.2, 72.5],
           [69.1, 72.3],
           [68.4, 72.3],
           [67.4, 72. ],
           [68. , 71.7],
           [68.3, 71.6],
           [70.3, 71.6],
           [70.7, 71.8],
           [69.6, 71.9],
           [68.6, 71.7],
           [69.6, 71.4],
           [69.3, 71.4],
           [67.5, 71.1],
           [67.6, 70.7],
           [68. , 70.9],
           [68.1, 71. ],
    ...
           [72.1, 74.8],
           [73. , 74.8],
           [72.9, 74.7],
           [71. , 74.2],
           [71.7, 74.3],
           [71.8, 74.2],
           [70.9, 74.2],
           [70.3, 73.9],
           [69.1, 73.4],
           [69. , 72.9],
           [70.3, 72.8],
           [71.7, 72.9],
           [71.6, 72.9],
           [70.6, 72.7],
           [69.6, 72.6],
           [68.1, 72.3],
           [69.4, 72.5],
           [70.6, 72.6],
           [69.6, 72.7],
           [70.8, 72.6]])

  • Record High Daily Average
    (Date, variable)
    float64
    78.0 80.4 77.8 ... 81.9 79.2 80.5

    array([[78. , 80.4],
           [77.8, 80.2],
           [78. , 80.7],
           [77.9, 81.2],
           [77.3, 81.4],
           [76.8, 81.7],
           [77.4, 81.5],
           [76.8, 78.8],
           [77.1, 78.2],
           [76.9, 78.2],
           [76.6, 78.2],
           [77.4, 77.2],
           [76.3, 77.4],
           [76.7, 77.8],
           [76. , 78. ],
           [76. , 78. ],
           [74.8, 77.4],
           [75.8, 78.6],
           [74.5, 79.8],
           [74.1, 80.4],
    ...
           [78.8, 81.7],
           [78.8, 82.6],
           [78. , 83.3],
           [78.4, 83.3],
           [78.4, 82.8],
           [79.1, 82.2],
           [78.8, 82.4],
           [78.9, 83.1],
           [78.3, 83.2],
           [77.6, 83.4],
           [78.4, 83. ],
           [79. , 83.2],
           [79.2, 83. ],
           [79. , 82.8],
           [78.6, 82.3],
           [78.4, 81.8],
           [79. , 82.2],
           [79.4, 82.6],
           [79.5, 81.9],
           [79.2, 80.5]])

  • Record High Daily Average Year
    (Date, variable)
    int64
    2022 2022 2022 ... 2016 2015 2021

    array([[2022, 2022],
           [2022, 2022],
           [2015, 2017],
           [2015, 2017],
           [2015, 2017],
           [2015, 2017],
           [2007, 2017],
           [2007, 2017],
           [2013, 2022],
           [2014, 2022],
           [2020, 2022],
           [2020, 2022],
           [2020, 2017],
           [2020, 2017],
           [2020, 2017],
           [2020, 2017],
           [2007, 2017],
           [2007, 2017],
           [2020, 2017],
           [2017, 2017],
    ...
           [2021, 2021],
           [2021, 2016],
           [2015, 2016],
           [2009, 2016],
           [2015, 2016],
           [2015, 2016],
           [2016, 2016],
           [2016, 2016],
           [2016, 2016],
           [2013, 2016],
           [2013, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2021]])

  • Record Low Daily Average
    (Date, variable)
    float64
    54.4 66.4 56.6 ... 66.4 49.9 66.1

    array([[54.4, 66.4],
           [56.6, 66.8],
           [51.7, 67.3],
           [51.6, 67.6],
           [51. , 66.2],
           [48.4, 64.8],
           [53. , 65.9],
           [49. , 66.2],
           [50. , 64.5],
           [54.6, 65. ],
           [59.6, 64. ],
           [52.5, 65.9],
           [53.6, 65.6],
           [58.6, 66.2],
           [61.2, 66.9],
           [58.4, 66.8],
           [55.4, 66.4],
           [50. , 66.7],
           [48.3, 64.9],
           [57.8, 65. ],
    ...
           [60.5, 69.1],
           [53.7, 67.6],
           [62.2, 66.3],
           [52.6, 65.8],
           [60.6, 63.7],
           [63.2, 65. ],
           [54.7, 65.3],
           [59.9, 67.1],
           [50.6, 67.1],
           [57.6, 65.8],
           [57.2, 66.8],
           [58.8, 67.8],
           [51.8, 67.3],
           [49.4, 66.4],
           [54.7, 66.6],
           [48.8, 65.8],
           [52.2, 65.2],
           [58.7, 66.6],
           [54.8, 66.4],
           [49.9, 66.1]])

  • Record Low Daily Average Year
    (Date, variable)
    int64
    2001 2011 2010 ... 2010 2000 2010

    array([[2001, 2011],
           [2010, 2011],
           [2012, 2011],
           [2018, 2001],
           [2010, 2001],
           [2010, 2001],
           [2010, 2010],
           [1996, 2002],
           [1996, 1996],
           [2001, 1996],
           [2004, 2010],
           [2010, 2010],
           [1996, 1996],
           [1996, 1996],
           [2012, 2011],
           [2014, 2011],
           [1997, 2012],
           [1997, 1997],
           [1997, 2003],
           [1997, 2003],
    ...
           [2004, 1996],
           [2010, 2010],
           [2017, 2010],
           [2010, 2010],
           [2010, 2010],
           [1997, 2010],
           [2003, 2010],
           [2003, 2010],
           [1996, 2003],
           [2009, 2003],
           [2012, 2003],
           [1995, 2010],
           [1995, 2003],
           [1995, 2003],
           [1995, 2010],
           [2010, 1995],
           [2010, 2010],
           [2009, 1995],
           [2000, 2010],
           [2000, 2010]])

  • Average High
    (Date, variable)
    float64
    75.0 73.7 75.1 ... 74.1 74.3 73.9

    array([[75. , 73.7],
           [75.1, 74.2],
           [73.9, 74.3],
           [73.2, 74.1],
           [72.9, 74. ],
           [73.9, 73.6],
           [73.4, 73.7],
           [72.2, 73.5],
           [72.3, 73.1],
           [73.4, 73.1],
           [74.1, 73. ],
           [74. , 73.2],
           [73.6, 73.2],
           [73.4, 73.1],
           [73.7, 72.9],
           [73.3, 73. ],
           [72.3, 72.5],
           [72.2, 72.2],
           [72.8, 72.3],
           [73.3, 72.4],
    ...
           [75.9, 75.9],
           [76.3, 75.9],
           [76.6, 75.9],
           [75.1, 75.7],
           [75.4, 75.4],
           [76.1, 75.3],
           [75.6, 75.4],
           [74.7, 75.3],
           [72.5, 74.6],
           [73.7, 74.2],
           [73.9, 74. ],
           [74.8, 74.3],
           [74.8, 74.2],
           [74.4, 74. ],
           [74. , 73.9],
           [72.7, 73.6],
           [73.1, 73.8],
           [74.4, 74.1],
           [73.2, 74.1],
           [74.3, 73.9]])

  • Lowest High
    (Date, variable)
    float64
    63.3 67.8 64.2 ... 68.4 55.6 67.8

    array([[63.3, 67.8],
           [64.2, 68. ],
           [57. , 68.9],
           [58.7, 69.1],
           [57.8, 68.5],
           [55.7, 66.2],
           [59.7, 67.1],
           [54.5, 67.8],
           [59.7, 66.6],
           [65.3, 67.3],
           [68.7, 68. ],
           [60.1, 68.9],
           [59.5, 68. ],
           [67.3, 67.6],
           [67.3, 68.9],
           [63.5, 68. ],
           [61.9, 67.5],
           [55.8, 68.2],
           [55.2, 67.1],
           [65.7, 66.9],
    ...
           [66.9, 69.8],
           [62.6, 70.9],
           [68.9, 69.8],
           [61.7, 69.3],
           [64.6, 65.3],
           [70.2, 68. ],
           [62.1, 68. ],
           [64.6, 68.9],
           [57. , 68.5],
           [63. , 68.7],
           [64. , 69.1],
           [62.2, 69.4],
           [57.9, 69.1],
           [57.7, 67.3],
           [61.7, 68.7],
           [55.4, 67.8],
           [57.7, 67.1],
           [67.1, 69.4],
           [59.5, 68.4],
           [55.6, 67.8]])

  • Lowest High Year
    (Date, variable)
    int64
    2001 2011 2010 ... 2010 2000 2010

    array([[2001, 2011],
           [2010, 2011],
           [2012, 2011],
           [2010, 2011],
           [2010, 2001],
           [2010, 2001],
           [2014, 2002],
           [1996, 2011],
           [1996, 1996],
           [2001, 1996],
           [2001, 2010],
           [2010, 2001],
           [1996, 1996],
           [1996, 2011],
           [2000, 1996],
           [2014, 2011],
           [1997, 2012],
           [1997, 2012],
           [1997, 2003],
           [1997, 2003],
    ...
           [2004, 1996],
           [2010, 2010],
           [2017, 2010],
           [2010, 2010],
           [1998, 2010],
           [1998, 2010],
           [2003, 2010],
           [2003, 1997],
           [1996, 1997],
           [2009, 2003],
           [1995, 2004],
           [1995, 2004],
           [1995, 2003],
           [1995, 2003],
           [1995, 2010],
           [2010, 1995],
           [1995, 1995],
           [2010, 2003],
           [2000, 2010],
           [2000, 2010]])

  • Record High
    (Date, variable)
    float64
    79.3 81.1 79.3 ... 83.1 80.1 81.7

    array([[79.3, 81.1],
           [79.3, 81. ],
           [79.2, 81.5],
           [81.7, 81.9],
           [80.2, 82.2],
           [78.3, 82.6],
           [81. , 82.2],
           [79.2, 80.6],
           [77.9, 79. ],
           [79. , 79.2],
           [79. , 79. ],
           [79. , 78.4],
           [78.4, 79.2],
           [78.4, 79.7],
           [78.6, 79.9],
           [80.4, 79.5],
           [77.7, 78.4],
           [77.7, 79.9],
           [78.4, 81.1],
           [80.2, 81.5],
    ...
           [81.5, 82.6],
           [82.9, 83.5],
           [80.1, 84. ],
           [80.4, 84. ],
           [79.7, 83.7],
           [82.2, 82.9],
           [82.8, 83.1],
           [79.9, 83.8],
           [80.4, 83.8],
           [79.3, 84. ],
           [80.4, 83.7],
           [79.9, 83.8],
           [80.1, 83.7],
           [79.7, 83.5],
           [80.8, 82.9],
           [80.2, 82.6],
           [80.1, 82.9],
           [80.6, 83.3],
           [80.6, 83.1],
           [80.1, 81.7]])

  • Record High Year
    (Date, variable)
    int64
    2022 2022 2022 ... 2016 2015 2021

    array([[2022, 2022],
           [2022, 2022],
           [2019, 2017],
           [2020, 2017],
           [2019, 2017],
           [2015, 2017],
           [2017, 2017],
           [2007, 2017],
           [2013, 2017],
           [2022, 2022],
           [2020, 2022],
           [2024, 2017],
           [2024, 2017],
           [2014, 2017],
           [2024, 2017],
           [1997, 2017],
           [2007, 2017],
           [2015, 2017],
           [2011, 2017],
           [2017, 2017],
    ...
           [2021, 2021],
           [2020, 2016],
           [2009, 2016],
           [2016, 2016],
           [2009, 2016],
           [2020, 2016],
           [2012, 2016],
           [2016, 2016],
           [2021, 2016],
           [2020, 2016],
           [2017, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2023, 2016],
           [2021, 2016],
           [2015, 2016],
           [2015, 2016],
           [2013, 2016],
           [2015, 2021]])

  • Average Low
    (Date, variable)
    float64
    67.9 71.1 68.5 ... 71.4 67.3 71.4

    array([[67.9, 71.1],
           [68.5, 71.6],
           [66.1, 71.7],
           [65.4, 71.3],
           [63.6, 71.1],
           [64.4, 71. ],
           [63.4, 70.9],
           [62.7, 70.6],
           [63.6, 70.4],
           [63.2, 70.1],
           [66.6, 70.2],
           [67.4, 70.5],
           [65.6, 70.6],
           [63.8, 70.3],
           [65.4, 70. ],
           [65.2, 70. ],
           [62.7, 69.8],
           [63. , 69.2],
           [63.2, 69.5],
           [62.8, 69.6],
    ...
           [68.3, 73.7],
           [69.7, 73.7],
           [69.2, 73.6],
           [66.9, 72.7],
           [68.1, 73.2],
           [67.5, 73.1],
           [66.3, 72.9],
           [65.8, 72.6],
           [65.8, 72.2],
           [64.3, 71.6],
           [66.6, 71.5],
           [68.5, 71.6],
           [68.3, 71.5],
           [66.7, 71.4],
           [65.3, 71.3],
           [63.4, 71. ],
           [65.8, 71.2],
           [66.7, 71.2],
           [66.1, 71.4],
           [67.3, 71.4]])

  • Highest Low
    (Date, variable)
    float64
    76.8 79.7 76.3 ... 80.8 78.3 79.3

    array([[76.8, 79.7],
           [76.3, 79.5],
           [77. , 79.9],
           [76.8, 80.6],
           [75.9, 80.6],
           [75.4, 80.8],
           [76.6, 80.8],
           [75.4, 77.9],
           [76.3, 77.5],
           [75.4, 77.2],
           [74.8, 77.5],
           [76.1, 76.6],
           [75.2, 75.9],
           [75. , 75.9],
           [74.3, 76.1],
           [74.5, 76.5],
           [73. , 76.5],
           [74.8, 77.2],
           [73. , 78.6],
           [70.5, 79.2],
    ...
           [76.1, 80.8],
           [77.5, 81.7],
           [76.5, 82.6],
           [77. , 82.6],
           [77. , 81.9],
           [78.1, 81.5],
           [77.4, 81.7],
           [77.9, 82.4],
           [77.4, 82.6],
           [76.3, 82.8],
           [77.4, 82.2],
           [78.1, 82.6],
           [78.3, 82.2],
           [78.3, 82. ],
           [77.2, 81.7],
           [77.5, 81.1],
           [77.9, 81.5],
           [78.3, 81.9],
           [78.4, 80.8],
           [78.3, 79.3]])

  • Highest Low Year
    (Date, variable)
    int64
    2022 2022 2017 ... 2016 2015 2021

    array([[2022, 2022],
           [2017, 2022],
           [2015, 2017],
           [2015, 2017],
           [2007, 2017],
           [2007, 2017],
           [2007, 2017],
           [2013, 2022],
           [2013, 2022],
           [2013, 2022],
           [2013, 2022],
           [2020, 2022],
           [2020, 2022],
           [2020, 2017],
           [2013, 2017],
           [2020, 2017],
           [2017, 2017],
           [2007, 2017],
           [2001, 2017],
           [2015, 2017],
    ...
           [1997, 2021],
           [2021, 2016],
           [2001, 2016],
           [2009, 2016],
           [2015, 2016],
           [2015, 2016],
           [2016, 2016],
           [2016, 2016],
           [2016, 2016],
           [2013, 2016],
           [2013, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2016],
           [2015, 2021]])

  • Record Low
    (Date, variable)
    float64
    45.5 63.9 49.1 ... 64.4 44.2 64.4

    array([[45.5, 63.9],
           [49.1, 64.4],
           [46.4, 65.5],
           [43.7, 64.5],
           [44.2, 63.6],
           [41. , 61.7],
           [45.5, 61.7],
           [43.5, 62.5],
           [40.3, 60.9],
           [43.9, 61.2],
           [49.3, 60.1],
           [44.9, 62.2],
           [46.4, 62.3],
           [50. , 62.2],
           [53.6, 63.9],
           [53.2, 65. ],
           [48.9, 65.3],
           [44.1, 64. ],
           [41.4, 62.8],
           [49.8, 63. ],
    ...
           [54.1, 67.3],
           [44.8, 64.4],
           [55.6, 62.8],
           [43.5, 62.2],
           [51.6, 62.1],
           [55.8, 62.1],
           [47.3, 62.6],
           [48.9, 63.7],
           [44.2, 65.1],
           [49.3, 63. ],
           [49.5, 64.4],
           [55.4, 64.9],
           [45.7, 65.5],
           [41. , 63.3],
           [47.5, 64. ],
           [42.1, 63. ],
           [42.4, 62.2],
           [50.1, 62.2],
           [50. , 64.4],
           [44.2, 64.4]])

  • Record Low Year
    (Date, variable)
    int64
    2001 2001 2010 ... 2010 2000 2010

    array([[2001, 2001],
           [2010, 2001],
           [2012, 2001],
           [2018, 2010],
           [2010, 2010],
           [2010, 2010],
           [2010, 2010],
           [1996, 2010],
           [1996, 2010],
           [2001, 2010],
           [2004, 2010],
           [2010, 2010],
           [2011, 2010],
           [1996, 2010],
           [2012, 2010],
           [2014, 2010],
           [1997, 2012],
           [1997, 1997],
           [1997, 2003],
           [1997, 1997],
    ...
           [2004, 2010],
           [2010, 2010],
           [2017, 2010],
           [2010, 2010],
           [2010, 2010],
           [1997, 2010],
           [2003, 2010],
           [1996, 2010],
           [1996, 2003],
           [2003, 2003],
           [2012, 2003],
           [1995, 2003],
           [1995, 2003],
           [1995, 1995],
           [2010, 1995],
           [2010, 2010],
           [2010, 2010],
           [2009, 2010],
           [2000, 2010],
           [2000, 2010]])

  • Years
    (Date, variable)
    int64
    23 24 23 24 23 ... 24 24 24 23 24

    array([[23, 24],
           [23, 24],
           [23, 24],
           [22, 24],
           [23, 24],
           [23, 24],
           [23, 24],
           [22, 24],
           [23, 24],
           [22, 24],
           [22, 24],
           [23, 24],
           [23, 24],
           [23, 24],
           [23, 24],
           [23, 24],
           [23, 24],
           [23, 24],
           [23, 24],
           [23, 24],
    ...
           [24, 24],
           [24, 24],
           [23, 24],
           [24, 24],
           [24, 24],
           [24, 24],
           [24, 24],
           [24, 24],
           [24, 24],
           [24, 24],
           [24, 24],
           [24, 24],
           [23, 23],
           [24, 24],
           [23, 23],
           [24, 24],
           [24, 24],
           [24, 24],
           [24, 24],
           [23, 24]])

• Indexes: (2)
  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Water Temperature'], dtype='object', name='variable'))

  • Date
    PandasIndex

    PandasIndex(Index(['01-Jan', '02-Jan', '03-Jan', '04-Jan', '05-Jan', '06-Jan', '07-Jan',
           '08-Jan', '09-Jan', '10-Jan',
           ...
           '22-Dec', '23-Dec', '24-Dec', '25-Dec', '26-Dec', '27-Dec', '28-Dec',
           '29-Dec', '30-Dec', '31-Dec'],
          dtype='object', name='Date', length=366))

• Attributes: (12)

  datum :

  MHHW

  day_threshold :

  2

  hr_threshold :

  3

  last_updated :

  2024-05-25 10:00:00

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  F

  Water Temperature units :

  F

  Air Temperature data range :

  ('1994-04-01', '2024-04-30')

  Water Temperature data range :

  ('1994-04-01', '2024-04-30')

monthly_records

<xarray.Dataset> Size: 3kB
Dimensions:                           (Month: 12, variable: 2)
Coordinates:
  * variable                          (variable) object 16B 'Air Temperature'...
  * Month                             (Month) object 96B 'Jan' 'Feb' ... 'Dec'
Data variables: (12/16)
    Monthly Average                   (Month, variable) float64 192B 68.7 ......
    Record High Monthly Average       (Month, variable) float64 192B 72.6 ......
    Record High Monthly Average Year  (Month, variable) int64 192B 2013 ... 2016
    Record Low Monthly Average        (Month, variable) float64 192B 63.0 ......
    Record Low Monthly Average Year   (Month, variable) int64 192B 2001 ... 2010
    Average High                      (Month, variable) float64 192B 76.0 ......
    ...                                ...
    Average Low                       (Month, variable) float64 192B 55.6 ......
    Highest Low                       (Month, variable) float64 192B 63.5 ......
    Highest Low Year                  (Month, variable) int64 192B 2013 ... 2016
    Record Low                        (Month, variable) float64 192B 48.3 ......
    Record Low Year                   (Month, variable) int64 192B 1997 ... 2010
    Years                             (Month, variable) int64 192B 23 24 ... 24
Attributes:
    datum:                         MHHW
    day_threshold:                 2
    hr_threshold:                  3
    last_updated:                  2024-05-25 10:00:00
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    tz:                            lst
    unit_system:                   english
    Air Temperature units:         F
    Water Temperature units:       F
    Air Temperature data range:    ('1994-04-01', '2024-04-30')
    Water Temperature data range:  ('1994-04-01', '2024-04-30')

xarray.Dataset

• Dimensions:
  • Month: 12
  • variable: 2
• Coordinates: (2)
  • variable
    (variable)
    object
    'Air Temperature' 'Water Tempera...

    array(['Air Temperature', 'Water Temperature'], dtype=object)

  • Month
    (Month)
    object
    'Jan' 'Feb' 'Mar' ... 'Nov' 'Dec'

    array(['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct',
           'Nov', 'Dec'], dtype=object)

• Data variables: (16)
  • Monthly Average
    (Month, variable)
    float64
    68.7 71.6 70.8 ... 77.4 71.4 74.0

    array([[68.7, 71.6],
           [70.8, 72.8],
           [72.3, 75. ],
           [75.6, 78.5],
           [78.7, 82. ],
           [81.5, 85.2],
           [82.9, 87.1],
           [83.2, 87.3],
           [82. , 85.7],
           [79.6, 82.1],
           [75. , 77.4],
           [71.4, 74. ]])

  • Record High Monthly Average
    (Month, variable)
    float64
    72.6 78.9 74.9 ... 81.5 76.9 82.5

    array([[72.6, 78.9],
           [74.9, 76.3],
           [77.6, 80. ],
           [79.4, 83.4],
           [80.7, 84.1],
           [83.6, 87.6],
           [85. , 89.5],
           [85.9, 90.1],
           [82.7, 89.5],
           [81.2, 85.9],
           [78.6, 81.5],
           [76.9, 82.5]])

  • Record High Monthly Average Year
    (Month, variable)
    int64
    2013 2017 2018 ... 2016 2015 2016

    array([[2013, 2017],
           [2018, 2021],
           [2003, 2003],
           [2020, 2020],
           [1995, 2021],
           [2010, 2010],
           [2023, 2023],
           [2022, 2021],
           [2017, 2021],
           [2020, 2021],
           [2015, 2016],
           [2015, 2016]])

  • Record Low Monthly Average
    (Month, variable)
    float64
    63.0 67.5 65.5 ... 74.6 62.1 68.2

    array([[63. , 67.5],
           [65.5, 67.7],
           [66.1, 69.2],
           [72.8, 75.1],
           [77. , 78.6],
           [79.8, 83.3],
           [81. , 84.5],
           [81.8, 85.4],
           [80.6, 82.7],
           [77.5, 79.6],
           [71.4, 74.6],
           [62.1, 68.2]])

  • Record Low Monthly Average Year
    (Month, variable)
    int64
    2001 2001 1996 ... 2012 2010 2010

    array([[2001, 2001],
           [1996, 2005],
           [2010, 2010],
           [2004, 2004],
           [2013, 2001],
           [2014, 2002],
           [2013, 2013],
           [1994, 1995],
           [2001, 2004],
           [2000, 2000],
           [2012, 2012],
           [2010, 2010]])

  • Average High
    (Month, variable)
    float64
    76.0 75.4 76.5 ... 80.6 77.5 77.6

    array([[76. , 75.4],
           [76.5, 76.7],
           [78.5, 79.2],
           [80.8, 81.9],
           [82.5, 85.1],
           [84.8, 87.9],
           [85.8, 89.1],
           [85.7, 89.4],
           [85.1, 88. ],
           [83.8, 85.6],
           [79.7, 80.6],
           [77.5, 77.6]])

  • Lowest High
    (Month, variable)
    float64
    73.0 70.5 74.2 ... 76.8 72.5 73.0

    array([[73. , 70.5],
           [74.2, 73.7],
           [74.2, 73.3],
           [77.3, 78.4],
           [80.8, 82.8],
           [82.8, 85.3],
           [84.2, 86.4],
           [84. , 87.2],
           [83.9, 85.2],
           [81. , 82.7],
           [76.9, 76.8],
           [72.5, 73. ]])

  • Lowest High Year
    (Month, variable)
    int64
    2011 2011 2000 ... 2001 2010 2003

    array([[2011, 2011],
           [2000, 2016],
           [2010, 2010],
           [2004, 2010],
           [2014, 2013],
           [2014, 1996],
           [2012, 2013],
           [2003, 2000],
           [2000, 1997],
           [2010, 2004],
           [2012, 2001],
           [2010, 2003]])

  • Record High
    (Month, variable)
    float64
    78.0 81.7 78.6 ... 85.0 79.6 84.6

    array([[78. , 81.7],
           [78.6, 81.5],
           [82.8, 83.2],
           [85.8, 85.8],
           [85.2, 87.7],
           [87.6, 90.4],
           [88.7, 92. ],
           [88.5, 92.2],
           [86.7, 91.2],
           [86.8, 89. ],
           [82. , 85. ],
           [79.6, 84.6]])

  • Record High Year
    (Month, variable)
    int64
    2015 2017 2021 ... 2020 1994 2016

    array([[2015, 2017],
           [2021, 2021],
           [2003, 2021],
           [2020, 2020],
           [1995, 2021],
           [2009, 2010],
           [2018, 2021],
           [2022, 2021],
           [2021, 2021],
           [2023, 2016],
           [2020, 2020],
           [1994, 2016]])

  • Average Low
    (Month, variable)
    float64
    55.6 67.6 59.4 ... 74.2 59.2 70.2

    array([[55.6, 67.6],
           [59.4, 68.9],
           [63.3, 70.9],
           [68.3, 74.6],
           [73.8, 78.1],
           [77.6, 82.2],
           [79. , 84.5],
           [79.3, 84.4],
           [78.2, 83. ],
           [72.7, 77.9],
           [66. , 74.2],
           [59.2, 70.2]])

  • Highest Low
    (Month, variable)
    float64
    63.5 75.9 70.0 ... 80.0 70.5 79.6

    array([[63.5, 75.9],
           [70. , 73.4],
           [72. , 75.4],
           [72.6, 81.1],
           [77.1, 80.4],
           [80.8, 85.2],
           [82.3, 87.2],
           [83.6, 87.2],
           [79.8, 87.1],
           [77.8, 83.4],
           [74.4, 80. ],
           [70.5, 79.6]])

  • Highest Low Year
    (Month, variable)
    int64
    2013 2017 2018 ... 2016 2015 2016

    array([[2013, 2017],
           [2018, 2023],
           [1997, 1997],
           [2015, 2020],
           [2003, 1994],
           [2004, 2004],
           [2022, 2023],
           [2022, 2021],
           [2009, 2021],
           [1995, 2021],
           [2020, 2016],
           [2015, 2016]])

  • Record Low
    (Month, variable)
    float64
    48.3 64.0 47.9 ... 70.8 48.8 63.7

    array([[48.3, 64. ],
           [47.9, 63.2],
           [55.1, 64.8],
           [61.2, 69.4],
           [67.9, 74.1],
           [75.1, 80. ],
           [76.1, 80.9],
           [76.1, 81. ],
           [74.3, 79.5],
           [64.6, 73.2],
           [57.4, 70.8],
           [48.8, 63.7]])

  • Record Low Year
    (Month, variable)
    int64
    1997 2003 1996 ... 2000 2010 2010

    array([[1997, 2003],
           [1996, 2005],
           [1996, 2010],
           [2009, 2003],
           [1999, 2001],
           [1995, 1996],
           [2013, 2013],
           [1996, 2012],
           [2001, 2004],
           [2005, 2005],
           [2006, 2000],
           [2010, 2010]])

  • Years
    (Month, variable)
    int64
    23 24 23 21 24 ... 23 23 24 24 24

    array([[23, 24],
           [23, 21],
           [24, 25],
           [24, 25],
           [21, 23],
           [20, 23],
           [25, 25],
           [24, 25],
           [24, 24],
           [23, 23],
           [23, 24],
           [24, 24]])

• Indexes: (2)
  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Water Temperature'], dtype='object', name='variable'))

  • Month
    PandasIndex

    PandasIndex(Index(['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct',
           'Nov', 'Dec'],
          dtype='object', name='Month'))

• Attributes: (12)

  datum :

  MHHW

  day_threshold :

  2

  hr_threshold :

  3

  last_updated :

  2024-05-25 10:00:00

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  F

  Water Temperature units :

  F

  Air Temperature data range :

  ('1994-04-01', '2024-04-30')

  Water Temperature data range :

  ('1994-04-01', '2024-04-30')

monthly_records.coords['variable']

<xarray.DataArray 'variable' (variable: 2)> Size: 16B
array(['Air Temperature', 'Water Temperature'], dtype=object)
Coordinates:
  * variable  (variable) object 16B 'Air Temperature' 'Water Temperature'

xarray.DataArray

'variable'

• variable: 2

• 'Air Temperature' 'Water Temperature'

  array(['Air Temperature', 'Water Temperature'], dtype=object)

• Coordinates: (1)
  • variable
    (variable)
    object
    'Air Temperature' 'Water Tempera...

    array(['Air Temperature', 'Water Temperature'], dtype=object)

• Indexes: (1)
  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Water Temperature'], dtype='object', name='variable'))

• Attributes: (0)

We can still choose one environmental variable at a time, but now we get all of the records and corresponding years:

monthly_records.sel(variable='Air Temperature').to_dataframe().drop('variable', axis=1)

	Monthly Average	Record High Monthly Average	Record High Monthly Average Year	Record Low Monthly Average	Record Low Monthly Average Year	Average High	Lowest High	Lowest High Year	Record High	Record High Year	Average Low	Highest Low	Highest Low Year	Record Low	Record Low Year	Years
Month
Jan	68.7	72.6	2013	63.0	2001	76.0	73.0	2011	78.0	2015	55.6	63.5	2013	48.3	1997	23
Feb	70.8	74.9	2018	65.5	1996	76.5	74.2	2000	78.6	2021	59.4	70.0	2018	47.9	1996	23
Mar	72.3	77.6	2003	66.1	2010	78.5	74.2	2010	82.8	2003	63.3	72.0	1997	55.1	1996	24
Apr	75.6	79.4	2020	72.8	2004	80.8	77.3	2004	85.8	2020	68.3	72.6	2015	61.2	2009	24
May	78.7	80.7	1995	77.0	2013	82.5	80.8	2014	85.2	1995	73.8	77.1	2003	67.9	1999	21
Jun	81.5	83.6	2010	79.8	2014	84.8	82.8	2014	87.6	2009	77.6	80.8	2004	75.1	1995	20
Jul	82.9	85.0	2023	81.0	2013	85.8	84.2	2012	88.7	2018	79.0	82.3	2022	76.1	2013	25
Aug	83.2	85.9	2022	81.8	1994	85.7	84.0	2003	88.5	2022	79.3	83.6	2022	76.1	1996	24
Sep	82.0	82.7	2017	80.6	2001	85.1	83.9	2000	86.7	2021	78.2	79.8	2009	74.3	2001	24
Oct	79.6	81.2	2020	77.5	2000	83.8	81.0	2010	86.8	2023	72.7	77.8	1995	64.6	2005	23
Nov	75.0	78.6	2015	71.4	2012	79.7	76.9	2012	82.0	2020	66.0	74.4	2020	57.4	2006	23
Dec	71.4	76.9	2015	62.1	2010	77.5	72.5	2010	79.6	1994	59.2	70.5	2015	48.8	2010	24

Finally, write these to file for safe keeping.

daily_records.to_netcdf(os.path.join(outdir, 'statistics-daily.nc'), mode='w')
monthly_records.to_netcdf(os.path.join(outdir, 'statistics-monthly.nc'), mode='w')

<frozen importlib._bootstrap>:241: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

We will plot these results in Part 3.

---